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ABSTRACT 

This paper encourages the use of Total Quality 
Management (TQM) tactics and other strategies as a practical means 
fostering a quality culture in institutional research offices of 
higher education institutions. In particular, four approaches are 
advocated for understanding and minimizing errors: (1) use of TQM 
tactical tools to enhance work processes and error trapping- (2) 
institutional research climate change tactics aimed at error 
correction, prevention, and quality enhancement; (3) quality 
promotion tactics for working with external organizations; and (4) 
national leadership activities for promoting professional standards, 
developing benchmarks, and offering training for those working in 
institutional research. Example "bloopers" are used to illustrate 
errors occurring in institutional research offices and ways to detect 
or avoid them. (Contains 29 references.) (JB) 
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Abstract 



Institutional research offices must produce quality work which is timely, contains zero 
defects, and is accurate and informative. To prosper under such pressures, 
institutional researchers must adopt effective strategies to ensure quality- This paper 
encourages use of total quality management tactics and other strategies as a practical 
means of fostering a quality culture. Example "bloopers" are used to illustrate errors 
occurring in institutional research offices and ways to detect or avoid them. Encouraging 
quality practices in other organizations on whom institutional research offices depend for 
data is recommended. The role of professional associations in setting standards, providing 
training, and promoting excellence is stressed. The paper should be of interest to all 
those concerned with quality institutional research work. 
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i m >. THE E*ROR OF OUR WAYS: 

USING TQM TACTICS TO COMBAT INSTITUTIONAL RESEARCH BLOOPERS 

Errors present an opportunity for professionals to learn about the inner workings of their own 
office and that o other offices on whidh they depend for data, project, or support They give a 

ZLT 7, m deSi8n ' " P ' e5enm °° «** 15 concre * e and IntitJet 

d^al fo d Per T' 0n ° f qUa ' i,y ° f 0U ' W ° rk - ** »<>«■ *» W°*hes at 
t r^TOM , 7 m ' nimi2in8 WhiCh "» in -'«°na, research work: 

eal h ' C *° 6nhanCe ^ Pr ° CKSeS W < 2 > — 

research climate change actios, aimed at error correction, prevention, and quality 

«rhancemen,; (3, Quality promotion tactics fe WQrklng ^ ^ ^ J 

(4) nahonal leadership activities for promoting professional standards, developing benchmarks 
-id offering training for those working in institutional research. 

UNDERSTANDING ERRORS 

It is almost as difficult to define W as . is to define "quality" Definitions of quality have 
focused on two main ideas: (I ) product performance or "fitness for use" Quran, 1 989 p. 1 5)- 

TJ^TTTT or " mee,ine cus,omers ' needs and reaso - bie 

Chlfee nd sh P e M «, * r KSeS are a ' SO imP0^,an, eValUa,i "8 3 P"*"- ^cording to 
Chafee and Sherr (1 992, p. v), ■ a quality pr0 cess means that all the steps within the 

rr;x^::" 6 from beginning ,o ^ work ^ ^ * — - 

in the area of professional services, product performance has ,o be judged in a larger context 
ban , to expectations of the customer. The quality of work of a medical Lor, for instance u 
normally ,udged by peer review. Accounting work is judged for conformance with profit 
tandards, such as those of me Financial Accounting Standards Board (FASB). As a EZj 
profession, there are no, ye, dear standards for work in the institutional research field bu, 1 1, 
individual can recognize and articulate what work is "good" or "substandard". For ampfe T 
survey which met the genera, expiations held by the adminisrraror who commissioned , ' 
could have methodological limitations in its sampling technique which would be v ewed a 
aw in the larger milieu of insritutlona, research professionals. So, customer safisfaZ a Z is 
ma adequate as a monitor of the quality of institutional research work 

wha! ri'lT- Tt " KrmS " Pr ° dUC ' Pr0dua m a " d «** ^.fsfaction, 

1 Z T\ „ , T CeP P, ° CKS " mitati0nS ' " " UnaCCep,aWe 10 a *« *>< one of four 
reasons: (1 ) inadequate conceptualization (understanding of the project goals or research 
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question was flawed); (2) poor design or method (an ineffective approach was taken to 
answering the research or project question); (3) flawed process (the strategies, tactics, or 
procedures chosen to implement the design for the product were flawed); or (4) poor 
presentation (the presentation or packaging of the product was ineffective). 

TOTAL QUALITY MANAGEMENT 

Programs for total quality assurance or management (TQM) have been implemented in 
higher education. Institutions like Oregon State University (Coate, 1990) and Delaware County 
Community College (DeCosmo et al, 1991) provide examples of major efforts to implement 
TQM campus-wide. A "full-blown" TQM approach requires extensive training for large numbers 
of staff and full commitment by top executives of a university or college to make it successful. It 
can take from 18 to 24 months before any real gains in productivity or profitability are apparent 
from the commitment (Berry,l991). Specialized TQN'i teams are often formed to attack specific 
problems as pilot demonstration projects, but the real goal of all such efforts is to instill a culture 
for quality within the organization so that day to day work styles and practices enhance quality. 
Transformation is the goal rather than just quality gains in target areas. 

There are two potential barriers to adopting TQM in institutional research offices: 
(1) lack of institutional commitment to TQM; and (2) size of the institutional research operation. 
Teeter and Lozier (1991) have correctly pointed out that departments in larger organizations, 
such as institutional research, can initiate TQM on their own, and that small-scale ground swell 
efforts may ultimately stimulate a wider institutional adoption of TQM. However, since 
institutional research offices often depend on services and data from the larger community, lack 
of external commitment to TQM can limit the effectiveness of a single office adopting a TQM 
approach. The size of an institutional research operation can also be a barrier to a full TQM 
effort. In a recent survey of institutional research offices in the Northeast, Volkwein (1990) found 
that nearly 80% of the 141 offices surveyed had fewer than 3 professional staff. Given the small 
size and heavy workload of most institutional research operations and offices, TQM is difficult to 
adopt as a specialized activity requiring extensive staff time. A more practical approach may be 
to integrate the use of tactical components of TQM into the work styles of institutional research 
offices. By adopting tactical applications of TQM techniques rather than the more long term 
strategic deployment of TQM planning, some gains in productivity and quality may be achieved 
by offices with less investment of scarce time and staff resources. In addition, by teaching and 
involving staff in TQM techniques, institutional research managers are at the same time fostering 
a culture of quality. Demonstrating that small scale efforts to improve quality are possible 
without a major TQM program can have a salutary effect on attitudes and approaches to quality 
in other offices as well. 
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INSTITUTIONAL RESEARCH CLIMATE 

Ellen Earle Chaffee has pointed out that "quality is a verb not a noun" (Chafee and Sherr, 
1992, p. 19) and that quality is not a landmark to be achieved but a process of continuous 
improvement Processes within an institutional research office are largely influenced by the 
institutional research climate, which is a critical factor for work quality. Tagiura and Litwin 
(1968, p. 1 1) describe how a person's environment has a large impact on the quality of work 
being done: 

'The way an individual carries out a given task depends upon what kind of person he is, on 
the one hand, and the setting in which he acts, on the other. Climate and many related 
terms such as environment, situation, conditions, and circumstances have been widely used 
to explain that a person or group can behave in very different ways, even when faced with 
similar tasks and problems." 

Tagiura and Litwin then define organizational climate as "the relatively enduring quality of the 
total internal environment of an organization that (a) is experienced by the members, 
(b) influences their behavior, and (c) can be described in terms of the values of a particular set of 
characteristics (or attributes) of the environment (Tagiura and Litwin, 1968, p, 27)." 

Litwin and Stringer (1968) stated the key components of organizational climate as 
(1) responsibility (degree of deler ition experienced by employees); (2) standards (expectations 
about work quality); (3) reward (recognition for good work versus disapproval for poor 
performance); (4) organizational clarity (orderliness versus disorderliness); and (5) team spirit 
(fellowship and trust within an organization). Quality organizations match up on these five 
points by stressing a high degree of meaningful delegation to employees or "entrusting" 
(Chaffee and Sherr, 1992), communicating the importance of quality in every part of the 
organization (Berry, 1991), recognizing and rewarding quality (Winter, 1991), having constancy 
of purpose (Deming, 1986), and encouraging teamwork (Chaffee and Sherr, 1992). 

The institutional research climate can be viewed as a subculture or counterculture within the 
larger organizational culture of the institution. Institutional research offices do not produce all 
of their work or their errors on their own; they are part of a larger institutional culture on which 
the office depends to establish requests and expectations for work, to define norms for work 
processes, and to obtain information, data and other resources. In addition, offices are part of a 
higher education community and a polity community in which resources, advice, and practices 
are shared. Thus, an institutional research office may, on any given project, obtain its data from 
national resources such as IPEDS, seek advice on methodology over a multi-national electronic 
network, obtain its analysis from a computer across campus, and prepare its presentation using 
slides obtained from a company in another part of the country. Communications or exchange 
with any one of these sources can contribute to quality work or to errors. 
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BLOOPER COLLECTION 

In order to examine the types of errors which occur in institutional research offices, two 
approaches were used. Fir§t, an error log (Table 1 ) was maintained by four offices for a two 
month period (two system offices, one institutional office, and one budget office). This log 
explored the nature, source, discovery, and possible avoidance of errors made within these 
offices over the study period. Second , institutional research professionals were approached by 
an electronic mail message, personal contact, and other means for additional blooper examples. 
Anonymity was assured to all respondents. 

The instructions for the error log asked participants to report any error which was not 
captured by normal error checking procedures. For example, a data entry error would not be 
reported unless the error was undetected through the normal checking patterns of the office or 
unless the error survived well beyond the project stage at which it occurred. All errors were 
reported by stage of work, as follows: (1) project definition (defining the work to be done); 
(2) project design (the plan for how to do the work); (3) project production (implementing the " 
work plan); (4) project presentation (the form and manner in which the work is delivered to the 
client); and (5) project evaluation (post-project review of the work). Participants were also asked 
to describe the stage of work and circumstances under which the error was discovered, the 
number of days between the error and its discovery, and whether the discovery was made by the 
institutional research office or an external source. In addition, participants were asked to provide 
any insights into why the error occurred and how it could be prevented in the future. 

The purpose of collecting the errors was to understand in detail how errors occur, are 
discovered, and avoided by institutional research professionals. Since the sample of errors for 
the study was not random, generalization of results to other institutional research offices is not 
warranted. The errors for the four offices participating in the error log, for instance, may be 
reflections of the manner in which those offices are organized, staffed, and resourced, or of the 
overall climate of the institutions in which the offices are imbedded. Errors reported by other 
offices may be unique or memorable errors, which do not occur regularly. The error collection 
in this study is intended only to provide examples of the types of errors which occur in 
institutional research offices and to make some suggestions for ways to deal with the types of 
errors reported. 

In all, a total of 51 errors were logged and reported. The errors were classified in the 
following categories: Conceptual errors (20%), design errors (18%), production errors (59%), and 
presentation errors (4%). Over 75% of the errors were discovered by institutional research staff 
rather than outsiders. Of all errors reported, though, nearly half escaped detection until after the 
project results had left the office. 
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Participants attributed half of all errors committed to problems during production. The 
specific problems included data entry (34%), data processing or management (31%), 
spreadsheet manipulation (19%), and computer programming (16%). Causes other than 
production problems were incorrect assumptions (16%), inadequate training or familiarity with 
the work being performed (1 1 %), time pressures (9%), poor design (8%), and political factors 
(6%). 

Forty percent of reporters indicated the best idea for prevention of errors was to carry out 
reasonableness checks of the results being produced. The reasonableness of the data could be 
verified through either an independent data source or someone with content knowledge of the 
data or policy area being studied. Improved communication was the next leading suggestion 
(24%), followed by better training or documentation (17%), improved process design (16%), or 
policy clarity (3%). 



TQM TACTICS 

At its core, TQM is a set of analytic techniques for identifying the needs of clients and the 
means by which the needs can best be met. Thus, TQM is particularly appropriate as a means of 
addressing conceptual errors in defining projects or process errors in conducting work, which 
represented nearly 80% of the errors reported. There are many good sources of information 
on TQM tools. The references by Brassard (1993), Chaffee and Sherr (1192), Stratton (1991), 
and Bossert (1991) all contain both discussion and examples of TQM tools. Three examples'of 
how to use TQM tools in diagnosing or preventing errors follow: 
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Example 1: Statistical Process Tools 

Bruce Beck, Associate Policy and Planning Analyst, University of Wisconsin, Madison, 
described a blooper which occurred because of the analysis of results of a survey done on racial 
climate by a faculty member at another institution. The survey design included deliberate over- 
sampling of some ethnic groups. The results of the initial analysis showed that UW-Madison's 
racial climate was rated worse than that of any of the other UW System institutions surveyed. 
This conclusion was picked up by the Wisconsin State Journal and the Milwaukee Sentinel and 
thus caught the attention of the Office of Budget, Planning and Analysis at UW-Madison. 

In order to examine the study results, Mr. Beck used two statistical process tools: an incidence 
checklist (Table 2) and a scatter plot (Table 3). These tools enabled him to see that the 
composition of the study sample was probably responsible for the results. Negative responses on 
racial climate were more prevalent among minority group members and UW-Madison had a 
higher incidence of minority respondents to the survey than all other institutions. He showed 
the study design was at fault in not presenting analyses separately for each category of ethnic 
status since some categories had been over-sampled. As an alternative, the weight given to 
responses could have been adjusted before making a summary comparison for racial climate by 
institution. In this example, the statistical process tools not only uncovered the design error, but 
also produced a powerful graphic which could then be used in communicating the real 
meaning of the study data. 

Example 2: Process Decision Program Chart 

Recently, a discrepancy was noted between state and national data on private high school 
enrollments for the State of New Hampshire. This discrepancy came to light as part of an effort 
to create enrollment projections for the University System of New Hampshire. The projections 
created by the Office of Policy Analysis differed considerably from data reported in a national 
study. The national study methodology called for using national data sources including the 
National Center for Educational Statistics (NCES) and the National Catholic Education 
Association (NCEA) on private high schools whenever state data was either unavailable or 
exceeded national data by a specific percentage. On the whole, this practice probably 
enhanced data quality as many states do not collect much data about private schools. The 
national study staff felt comfortable with this approach because of the high response rates 
reported by the NCES and NCEA. However, in some states, such as New Hampshire, this 
practice resulted in greatly over-estimating private school enrollments. The data from NCES 
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Table 2. Incidence Checklist: Survey Response 
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Table 3* Scatterplot: Response Characteristics and Racial Climate 
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appeared to account for the most variance between population data from the New Hampshire 
Department of Education and the national study estimates. The New Hampshire Department of 
Education data for 1989-90 included 162 private schools, with a total enrollment of 18,944. 
These data were known to represent the entire population since all private schools had to be 
licensed by the Department in order to operate. Enrollments were reported as part of the 
licensing process. The NCES data reported on a total of 144 schools, and estimated enrollments 
at 21,742 students, with a reported response rate of 97%. 

With the help of a Process Decision Program Chart, shown as Table 4, the mystery cleared. 
The NCES used a list frame of 109 schools (obtained from a vendor, not from the New 
Hampshire Department of Education) as the basis of the study. Then, a regional area frame 
sample was used to supplement the list frame. The area frame included two counties from New 
Hampshire, with 3 schools identified. The data from these 3 schools were then weighted up to 
represent 35 schools, based on the expected under-reporting from the list frame for the region. 
Otherwise said, the area frame survey had a standard error of 35 schools! The response rate for 
the NCES survey was reported as 97%, based on the response rate of the schools identified 
through the original list frame and the area frame. However, this "97%" response rate clearly did 
not represent 97% of the private schools in New Hampshire. Only 106 out of the population of 
1 62 schools actually provided data for the survey, for a "real" response rate of 65%. The rest of 
the data were imputed. (In fairness to NCES, there were clear caveats associated with these data 
indicating that the data should not be used for state estimates). Enrollments for the NCES survey 
were over-estimated by 15%, which then affected the validity of all subsequent uses of the data. 
NCES has taken steps to improve the processes for identifying schools for the list frame, and the 
national study team is reviewing its use of different data sources. 

Use of the Process Decision Program Chart was helpful in this example in three ways: (1 ) the 
chart specified the way in which the data estimates had been made; (2) decision rules and their 
consequences were clarified; and (3) the chart enabled clear discussion of process improvements 
for future research. 

Example 3: Cause and Effect and Matrix Diagram 

During collective bargaining, a dispute arose between faculty and the administrative 
bargaining team over the accuracy of salary analyses of executive salaries which had been 
presented by the faculty. The collective bargaining team assumed that the faculty had included 
data for the wrong data year, since the data included both salaries and some non-salary benefits. 
A response to the union report was prepared under that assumption. In reality, the budget 
office of the institution had prepared an earlier report which showed both salary and non-salary 
amounts as "salary". The confusion had resulted scnce a policy change had recently been made 
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Table 4. Process Decision Program Chart: 
Private School Enrollments, 1989-90 
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concerning executive compensation. Non-salary benefits which were specific only to principal 
administrators (PAs) were being rolled into salary over a two year period. The union had used 
the data from the budget report as the source for its own analysis. 

A modified cause and effect chart was constructed to identify factors which could have 
caused the problem, shown in Table 5. The form of the cause and effect diagram was adapted 
to better suit the institutional research environment. Categories were included for both process 
and cultural causes. The process category was broken down into definition, design, production, 
presentation, and evaluation causes. The cultural category included experience, standards, 
polity, resources, and communications. Using this chart, the main factors explaining the salary 
reporting problem were identified as policy concerns. For the data year in question, two 
different methods had been used to calculate salaries, one showing salaries only, and one 
showing salaries plus the non-salary benefits. The policy inconsistency on salary reporting was 
directly linked for the confusion over the source of the union's salary figures for executives. 

To further clarify the reporting issues, a matrix diagram (Table 6) was used. The diagram 
showed there were at least three different offices, under two different administrators, which 
were involved in producing salary reports. Documenting which offices and people needed to 
be informed of, or involved in, the policy decision spotlighted the need for additional 
communication links to avoid the problem in the future. 

■'iiimfijiiiTlfl^ — taaaaji — --• ■ ■• - - - . 
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Table 5. Cause and Effect Chart on Salary Reporting 
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Table 6: Matrix Diagram on Salary Reporting 
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CLIMATE CHANGE TACTICS FOR INSTITUTIONAL RESEARCH OFFICES 

Climate change in the institutional research office is the best long term strategy for quality 
improvement. Many aspects of climate change can be incrementally encouraged by managers. 
Three tactics are recommended: (1 ) systematic emphasis on quality; (2) use of Quality Action 
Questions; and (3) error analysis. 

Systematic Emphasis on Quality 

Teeter and Lozier (1991) have emphasized the need to be systematic and scientific in the 
approach to institutional research work. They recommended the use of modeling of work How, 
TQM tools, and attention to problem identification and analysis. Heverly (1993) described how 
to apply TQM to the daily operations of institutional research staff, recommending the use of 
flowcharts to identify and display steps in regularly scheduled projects, documentation of work 
requests, and encouragement of a Kaizen (or quality-oriented) attitude. The use of the 
Shewhart Tlan-Do-Check-Act" cycle was recommended by McLaughlin and Snyder (1993) in 
their discussion of the organization of institutional research offices, projects, and activities. 

The common theme of all these researchers is to systematically apply standards, techniques, 
and processes which emphasize quality. Whatever strategies or techniques are adopted should 
be used consistently, communicated clearly, and become part of the office reward structure. 

Quality Action Questions 

Nearly 60% of the errors reported were not caught before being passed on to the next stage 
of work. This indicated the need for a tool or strategy to help catch error.* during the work 
process as they occur, rather than later. A Quality Action Questions (QAQ) Checklist (shown in 
Table 7) was created for this purpose. Ideas for specific questions came from the blooper reports 
as well as from the general TQM literature. These questions can serve as a checklist for 
reviewing work at each stage of a work process - project definition, project design, project 
production, project presentation, and project evaluation. Two additional sections are included, 
one for working with external data sources and one for project debriefing, or those activities 
related to the close or "putting to bed" of a project. The QAQs imply and insure greater team 
and client involvement in all phases of the project work cycle. The advantage of using a quality 
checklist of this nature is the emphasis on quality during all stages of a research project, not just 
in evaluating the final product. 

Error Analysis 

Although it is always better to build quality processes than to spend time finding or inspecting 
for errors, much can be learned from each error which occurs. It is important to establish a non- 
threatening environment for the discussion of bloopers. Deming (1 986) called this removing 
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Table 7. Quality Action Questions (QAQs) Checklist 
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Project Definition 

Do all parties agree on the definition and scope of the project? 

2. Have all relevant areas been involved in the project definition? 

3. Do all parties understand and agree to the project timetable and resource requirements? 

4. Is there a dear outcome or product defined for the project? Presentation format? 

5. Do all parties agree on the outcome or product for the project? Its presentation format? 

6. Are the needs of the audience for the product understood? 



Project Design 



1. 



Has relevant work of other researchers been identified and reviewed? 

Do all members of the production team understand and agree with the study design and method? 
Have all data resources been identified and accessibility checked? 

4. Can the design produce a reasonable answer to the project question? 

5. Will the design produce the agreed-upon product in an efficient way? 



Project Production 

1 . Is the sequence of activities needed to produce the product clear and do-able? 

2. Is the project schedule realistic for the production team? 

3. Are all resources needed available for production? 
Do intermediate project results pass a reasonableness test? 
Do I get the same results using an independent data source, process, or method? 



External Data Resources 



Do I know all of the assumptions built into the data? 

2. Have I provided a clear statement of the definition and format for the data being requested? 

3. Will the data be delivered on a compatible medium? 
What procedures were followed by the data source in handling missing or incomplete data? 
Have the data been verified by the data source before delivery? 



Project Evaluation/Review 

1. Is the conclusion reasonable? 

2. Is the conclusion reasonable to someone familiar with the content area? 

3. Is the conclusion consistent with historical data? 

4. Is the conclusion consistent with what others have found doing similar work? 

5. Is the conclusion consistent with results from related areas? 



Project Presentation 

1 . Have the clients reviewed and approved a draft of the presentation product? 

2. Have audience requirements for the product presentation been met? 

3. Has the presentation been pretested on site (If involving media)? 

4. Is the presentation clear and understandable? 

5. Are visual or other media used appropriately in the presentation? 



Project DeBriefing 

1 . Have ail files, resource materials, and databases been put into a maintenance state? 

2. Have any insights for improvement been identified and recorded for later work? 

3. Has the client been asked for an evaluation of the project? 

4. Have you toasted or roasted the project in good humor and prepared staff to move on? 

5. Have you thanked all those who contributed to the project? 
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"fear" from your organization. As long as fear prevails, team members will not be able to fully 
participate in any improvement process, especially one focused on a potentially sensitive area 
such as errors. Some ideas for desensitizing the process of discussing errors include: (1) being 
honest and open about your own errors; (2) communicating that errors generally flow from work 
processes, not from the failure of an individual to do work well; (3) stressing that all members of a 
team "own" an error which occurs; (4) routinely using a team approach to examine and unravel 
the causes of error; and (5) stressing that errors are an opportunity for making change and 
improvements. 

Errors can also be analyzed by individuals on their own. Keeping an error log can serve to 
focus thinking about the causes of an error. Individuals can use the log, or a similar tool, in an 
entirely private way to examine and understand their own errors. Team discussion is more likely 
to result in process improvements for the entire office, though, since individuals generally have 
to work with and depend upon others to complete a complex project. 

QUALITY PROMOTION STRATEGIES FOR EXTERNAL ORGANIZATIONS 

Institutional research offices are very dependent on the data quality of other offices. Nearly 
half of all errors reported occurred because of a data quality lapse or operating assumption in an 
office external to the institutional research office. There are two types of common situations 
which can result in difficulty with external data: (1) sensitivity problems with national or regional 
studies; and (2) local data administration problems. 

Sensitivity Problems With National or Regional Studies 

Studies where an organization reports data for an entire region or country can be subject to 
sensitivity problems. In such studies, operating assumptions are made concerning missing data, 
which data sources to use, and data imputation which can markedly affect the accuracy of data 
for a particular state or institution. 

An example of an imputation problem was shared by Mark Chisholm, Senior Information and 
Research Officer for the Colorado Commission on Higher Education. In reporting on Fall 
Enrollment data for 1992, NCES imputed all unknown data on ethnic status into the five known 
ethnic categories, excluding the non-resident alien category. With respect to the Colorado 
data, NCES matched back to the wrong total, thus resulting in an error of as many as 500 
students at one Colorado school. Mr. Chisholm questioned both the procedural error which 
had been made, and the underlying assumption that students in all ethnic categories were 
equally likely to not respond to a question about ethnic status. 

Since we all depend upon the results of large scale studies, it is worth exploring and 
understanding the types of assumptions which have been made by the study directors and how 
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those assumptions may affect the results for your particular state or institution. Imputation 
practices should be based on supporting research rather than historical practice. 

In almost all cases, sensitivity about the reasonableness of study results and the meaning of 
particular data will be better for those closer to the data than for those organizing data on a 
national basis. With national studies, decisions made in the course of compiling the data can not 
be too "particularistic" or driven by the unique needs of one or more states or institutions. Yet, 
some national studies protect against errors in local data by building in a feedback loop for an 
institution (or state) to sign off on data accuracy. The American Association of University 
Professor's annua! salary study, managed by Maryese Eymonerie Consultants, is a notable 
example. Such a practice would have avoided the data errors described in this paper on 
enrollment projections and ethnicity reporting. In the absence of a feedback loop, it is often 
possible to receive copies or files of the data specific to your state or institution after the study has 
been released. While this is too late to prevent errors, following up on any discrepancies or 
problems seen can help prevent future errors. Most stucV leaders are very interested in such 
follow up questions and will work with institutional researchers to resolve problems in the future. 

Local Data Administration Problems 

In the absence of a clear data administration function on a campus, the value placed on 
particular data by different organizations may vary. Often members of institutional research 
offices care more about certain data elements than staff in other offices do. For example, at one 
institution, the Institutional Research Office used class codes in doing enrollment projections. In 
one year, the Registrar's Office, which "owned" the class codes was involved in a major software 
implementation. Since the class codes were not needed for any operational purposes in the 
Registrar's Office, maintenance on the field was not done for the year. As a consequence, the 
enrollment projections being created were wrong. In a similar example, the Institutional 
Research Office of another institution had been working on a housing study for married students. 
The proportion of married students appeared to be rapidly declining over the previous five years, 
a revelation of some alarm since the purpose of the study was to project needs for new student 
housing. On investigation, the reason for the decrease was that the Graduate School had 
stopped collecting data on the marital status of students, but had not removed the data element 
from the database. The "decrease" was actually caused by the graduation or attrition of those 
students for whom data had been previously captured! 

Several strategies can be used to proi.iote better data administration: (1) communicate 
carefully about the data you are using from each source; (2) spend some time getting familiar 
with the people and data management practices at each organization which vendors data to 
your office; (3) create a data sharing committee to review practices and policies concerning data 



18 



The Error of Our Ways 
15 

maintenance, access, and use within your institution; (4) promote the need for data 
administration in the form of either a position or an organized work group at your institution; 
(5) promote training for all those managing data on which you depend (even if you have to offer 
the training yourself); (6) build partnerships with other organizations who provide and use the 
data on which your office depends; and (7) promote better understanding of data quality issues 
among the leaders of your institution. 

NATIONAL LEADERSHIP ACTIVITIES 

While individual institutional research offices are primarily responsible for achieving quality 
work in their own institutional setting, professional organizations such as the Association for 
Institutional Research (AIR) and The Society for College and University Planning (SCUP), can 
promote quality in three ways: (1) developing standards; (2) conducting benchmarking studies; 
and (3) organizing and providing professional training. 

Standards 

AIR members have recently developed a code of ethics about the way in which we go about 
our work, but there are few real standards in place for what we do. A recent study by a state 
higher education commission provides an example of how methodological standards could 
facilitate policy analysis and discussion. Table 8 shows an analysis by the commission on faculty 
workload. The analysis reported the number of faculty and current number of courses taught at 
each public institution in the state system. The conclusion of the analysis was that the public 
institutions in the state would be able to teach nearly 4,000 additional courses without any 
funding increase, or that the institutions could cut roughlv 550 faculty systemwide without a drop 
in service to students. Caveats were included in the report about the need to collect additional 
data on faculty workload. Officials of the higher education system in the state responded (prior 
to the study's publication) that the study did not take into account class size, type of course, 
course credits, course discipline, or level of instruction, and thus did not present an accurate 
assessment of actual workload for faculty. The commission published its results without 
modification. In this case, both the commission and system organization believed their 
operationalization of faculty workload to be appropriate. In the absence of methodological 
standards for research on faculty workload, no agreement was reached on the issue of 
interpretation of data. Consequently, no common base of understanding was developed which 
could frame discussion of the larger policy, issues on workload. It is critical to have well-defined 
methodological standards for research in areas where a public policy debate is engaged. 
Otherwise, neither the general public nor policy makers are positioned to both understand and 
debate solutions to problems. 
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Table 8. State Commission Analysis of Faculty Workload 



Cora Faculty 1991*92 
Current Versus Potential Courseioad 


Carnegie 
Class. 


Total 
Core 
Faculty 


Current 
Course 
Load 


roieniiai 

Load 
Increase* 


Potential 

Course 
Loaa 


PcL 
inc. 


Research 1 












A 


1 052 




1,737 


5,421 


47% 


Research II 












B 


271 


1,155 


314 


1,469 


27% 


ComDrehensive I 












c 


82 


603 


80 


683 


13% 


D 


222 


1,579 


268 


1347 


17% 


£ 


189 


1,355 


194 


1,549 


14% 


F 


437 


2,905 


704 


3,609 


24% 


G 


118 


597 


351 


£48 


59% 


Comprehensive II 












H 


94 


600 


175 


775 


29% 


S 


55 


404 


63 


467 


16% 


Toul 


2,520 


12,882 


3,886 


16,768 


30% 



/ 



• Assumes the 1 991 -92 core faculty number and a standard 
counebad of eight courses a year at comprehensive institutions 
and five courses per year at the research institutions* 



AIR has sponsored some focused work on research standards. For example, the Data Advisory 
Committee of AIR has worked on reporting standards for the Student Right-to-Know and 
Campus Security Act of 1990. There are models for how to begin developing a breeder code of 
written standards. The government, accounting, and business officer associations such as The 
National Association of College and University Business Officers (NACUBO) have developed 
standards for financial management. NCES (1992) has recently published extensive statistical 
standards for research. These standards cover planning and testing, contract management and 
operations, statistical processing procedures, data provisions and analysis, reporting, and 
evaluation and documentation for surveys. 

Many other professions have in place formalized review or certification structures based on 
peer review. Examples are the accreditation reviews of academic programs or financial audits of 
business organizations 1 practices. Lack of formal standards hinders the professionalization and 
limits the respect given an emerging field such as institutional research. A voluntary 
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accreditation or certification process might encourage and recognize those offices with high 
standards for quality. AIR has studied the idea , but is not currently pursuing development of a 
certification program. 

Benchmarking 

A related strategy for strengthening institutional research operations could be copied from 
the latest efforts of NACUBO to benchmark the costs of many operational and administrative 
services at a variety of institutions. According to Shafer and Coate, "bench marking is an ongoing, 
systematic process for measuring and comparing the work processes of one organization to 
another. It identifies 'best practices' that can lead to improvements in operations and customer 
service.'' It is a "positive, proactive structured process which leads to changing operations and 
eventually attaining superior performance" (Camp, 1989, p. xi). The underlying philosophy of 
benchmarking is captured in the Japanese word "dantotsu" or striving to be the best of the best 
(Camp, 1989). 

Benchmarking protocols or studies could help institutional research offices to gauge their 
effectiveness on specific tasks and provide a baseline from which to track improvements in 
quality management and operations. Benchmarking enables an office to gauge its "batting 
average" against that of other offices (or against itself over time), to identify specific areas for 
improvement, and to define and measure improvements after initiatives have been taken. 
Some common activities in most institutional research offices could serve as subjects for 
benchmarking, including IPEDS reports, the AAUP annual faculty salary survey, or the graduation 
rate reporting which will be required under the Student Right to Know and Campus Security 
Act of 1990. 

Training 

The development of an institutional research curriculum is a third area in which associations 
could assist professionals in fostering quality work. To its credit, AIR has adopted training as a key 
emphasis in its strategic planning process. The institute held in the summer of 1993 (and 
planned for August 1994) is a program designed to meet *he training needs of beginning 
institutional research professionals. According to AIR Executive Director, Terrence Russell, in 
1995, the institute program will include both a general institute similar to those offered 
previously and a new advanced institute. AIR is also developing a program to cost-share training 
with regional organizations willing to use the training obtained to provide workshops. Other 
AIR training initiatives include the pre-conference workshops associated with annual meetings, 
courseware development, and publication of materials such as the planned Resources in 
Institutional Research series. 
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CONCLUSIONS 

Institutional research offices strive to produce quality work which is timely, contains zero defects, 
is accurate and is informative. Bloopers reported by institutional researchers were analyzed and 
used as examples to show four different approaches to preventing errors: (1) using total quality 
management tools to diagnose and prevent errors; (2) promoting a quality culture in 
institutional research offices through systematic emphasis on quality, use of Quality Action 
Questions, and error analysis; (3) encouraging better data management and administration in 
offices external to institutional research; and (4) providing leadership through activities of 
national associations to set standards, provide training, and promote excellence. Institutional 
research does not occur in an organizational vacuum. Quality improvements depend partly 
upon the ability of the individual office to design and produce good work, and partly on the 
existence of a quality culture in the immediate environment and the larger professional milieu. 
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